Systems and methods of automatically processing electronic images across regions

ABSTRACT

Systems and methods are disclosed for using an integrated computing platform to view and transfer digital pathology slides using artificial intelligence, the method including receiving at least one whole slide image in a cloud computing environment located in a first geographic region, the whole slide image depicting a medical sample associated with a patient, the patient being located in the first geographic region; storing the received whole slide image in a first encrypted bucket; applying artificial intelligence to perform a classification of the at least one whole slide image, the classification comprising steps to determine whether portions of the medical sample depicted in the whole slide image are healthy or diseased; based on the classification of the at least one whole slide image, generating metadata associated with the whole slide image; and storing the metadata in a second encrypted bucket.

RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Application No.62/989,095 filed Mar. 13, 2020, the entire disclosure of which is herebyincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally tocomputational pathology devices for processing electronic images. Morespecifically, particular embodiments of the present disclosure relate tocomputational pathology devices configured to build clinical-gradeproducts for the treatment of cancer. The present disclosure furtherprovides systems and methods for diagnostic accuracy, reliability,efficiency and accessibility.

BACKGROUND

The present disclosure describes computational pathology processes anddevices configured to be used to build clinical-grade products that maytransform the diagnosis and treatment of cancer. By using computationalpathology, pathologists may improve their diagnostic accuracy,reliability, efficiency, and accessibility. For example, a device maydetect slides as being suspicious for cancer, allowing pathologists tocheck their initial assessments before rendering a final diagnosis.Computational pathology processes and devices of the present disclosuremay provide an integrated platform allowing the ingestion, processing,and viewing of digital pathology images via a web-browser, whileintegrating with a Laboratory Information System (LIS).

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of thedisclosure. The background description provided herein is for thepurpose of generally presenting the context of the disclosure. Unlessotherwise indicated herein, the materials described in this section arenot prior art to the claims in this application and are not admitted tobe prior art, or suggestions of the prior art, by inclusion in thissection.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for using an integrated computing platform to viewand transfer digital pathology slides using artificial intelligence(AI).

A method for using an integrated computing platform to view and transferdigital pathology slides using artificial intelligence (AI) includes:receiving at least one whole slide image in a cloud computingenvironment located in a first geographic region, the whole slide imagedepicting a medical sample associated with a patient, the patient beinglocated in the first geographic region; storing the received whole slideimage in a first encrypted bucket; applying artificial intelligence toperform a classification of the at least one whole slide image, theclassification comprising steps to determine whether portions of themedical sample depicted in the whole slide image are healthy ordiseased; based on the classification of the at least one whole slideimage, generating metadata associated with the whole slide image; andstoring the metadata in a second encrypted bucket.

A system for using an integrated computing platform to view and transferdigital pathology slides using artificial intelligence (AI) includes: amemory storing instructions; and at least one processor executing theinstructions to perform a process including receiving at least one wholeslide image in a cloud computing environment located in a firstgeographic region, the whole slide image depicting a medical sampleassociated with a patient, the patient being located in the firstgeographic region; storing the received whole slide image in a firstencrypted bucket; applying artificial intelligence to perform aclassification of the at least one whole slide image, the classificationcomprising steps to determine whether portions of the medical sampledepicted in the whole slide image are healthy or diseased; based on theclassification of the at least one whole slide image, generatingmetadata associated with the whole slide image; and storing the metadatain a second encrypted bucket.

A non-transitory computer-readable medium storing instructions that,when executed by a processor, cause the processor to perform a methodfor using an integrated computing platform to view and transfer digitalpathology slides using artificial intelligence (AI), the methodincluding: receiving at least one whole slide image in a cloud computingenvironment located in a first geographic region, the whole slide imagedepicting a medical sample associated with a patient, the patient beinglocated in the first geographic region; storing the received whole slideimage in a first encrypted bucket; applying artificial intelligence toperform a classification of the at least one whole slide image, theclassification comprising steps to determine whether portions of themedical sample depicted in the whole slide image are healthy ordiseased; based on the classification of the at least one whole slideimage, generating metadata associated with the whole slide image; andstoring the metadata in a second encrypted bucket.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory only,and are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 is an exemplary global architecture of a platform for processingdigital slides, according to an exemplary embodiment of the presentdisclosure.

FIG. 2 is a flowchart illustrating an exemplary method for use of theplatform, according to an exemplary embodiment of the presentdisclosure.

FIG. 3 is a workflow illustrating an exemplary method for use of theplatform with an artificial intelligence (AI) output, according to anexemplary embodiment of the present disclosure.

FIGS. 4A-C are exemplary architectures of a data ingestion appliance andintegrations of the data ingestion appliance, according to exemplaryembodiments of the present disclosure.

FIGS. 5A-C are exemplary architecture of a LIS and integration of theLIS, according to exemplary embodiments of the present disclosure.

FIG. 6 is an exemplary architecture of a slide viewer, according to anexemplary embodiment of the present disclosure.

FIG. 7 is an exemplary architecture of an AI computer, according to anexemplary embodiment of the present disclosure.

FIGS. 8A-C are exemplary displays from the slide viewer.

FIG. 9 depicts an example system that may execute techniques presentedherein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Pathology refers to the study of diseases. More specifically, pathologyrefers to performing tests and analysis that are used to diagnosediseases. For example, tissue samples may be places onto slides to beviewed under a microscope by a pathologist (e.g., a physician that is anexpert at analyzing tissue samples to determine whether anyabnormalities exist). That is, pathology specimens may be cut intomultiple sections, prepared as slides, and stained for a pathologist toexamine and render a diagnosis. When uncertain of a diagnostic findingon a slide, a pathologist may order additional cut levels, stains, orother tests to gather more information from the tissue. Technician(s)may then create new slide(s) which may contain the additionalinformation for the pathologist to use in making a diagnosis. Thisprocess of creating additional slides may be time-consuming, not onlybecause it may involve retrieving the block of tissue, cutting it tomake a new a slide, and then staining the slide, but also because it maybe batched for multiple orders. This may significantly delay the finaldiagnosis that the pathologist renders. In addition, even after thedelay, there may still be no assurance that the new slide(s) will haveinformation sufficient to render a diagnosis.

Pathologists may evaluate cancer and other disease pathology slides inisolation. The present disclosure presents a platform for improvingdiagnosis of cancer and other diseases. The platform may integrate, forexample, slide evaluation, tasks, image analysis and cancer detectionartificial intelligence (AI), annotations, consultations, andrecommendations in one workstation. In particular, the presentdisclosure describes various exemplary user interfaces available in theplatform, as well as AI tools that may be integrated into the platformto expedite and improve a pathologist's work.

For example, computers may be used to analyze an image of a tissuesample to quickly identify whether additional information may be neededabout a particular tissue sample, and/or to highlight to a pathologistan area in which he or she should look more closely. Thus, the processof obtaining additional stained slides and tests may be doneautomatically before being reviewed by a pathologist. When paired withautomatic slide segmenting and staining machines, this may provide afully automated slide preparation pipeline. This automation has, atleast, the benefits of (1) minimizing an amount of time wasted by apathologist determining a slide to be insufficient to make a diagnosis,(2) minimizing the (average total) time from specimen acquisition todiagnosis by avoiding the additional time between when additional testsare ordered and when they are produced, (3) reducing the amount of timeper recut and the amount of material wasted by allowing recuts to bedone while tissue blocks (e.g., pathology specimens) are in a cuttingdesk, (4) reducing the amount of tissue material wasted/discarded duringslide preparation, (5) reducing the cost of slide preparation bypartially or fully automating the procedure, (6) allowing automaticcustomized cutting and staining of slides that would result in morerepresentative/informative slides from samples, (7) allowing highervolumes of slides to be generated per tissue block, contributing to moreinformed/precise diagnoses by reducing the overhead of requestingadditional testing for a pathologist, and/or (8) identifying orverifying correct properties (e.g., pertaining to a specimen type) of adigital pathology image, etc.

The process of using computers to assist pathologists is known ascomputational pathology. Computing methods used for computationalpathology may include, but are not limited to, statistical analysis,autonomous or machine learning, and AI. AI may include, but is notlimited to, deep learning, neural networks, classifications, clustering,and regression algorithms. By using computational pathology, lives maybe saved by helping pathologists improve their diagnostic accuracy,reliability, efficiency, and accessibility. For example, computationalpathology may be used to assist with detecting slides suspicious forcancer, thereby allowing pathologists to check and confirm their initialassessments before rendering a final diagnosis.

Histopathology refers to the study of a specimen that has been placedonto a slide. For example, a digital pathology image may be comprised ofa digitized image of a microscope slide containing the specimen (e.g., asmear). One method a pathologist may use to analyze an image on a slideis to identify nuclei and classify whether a nucleus is normal (i.e.,benign) or abnormal (i.e., malignant). To assist pathologists inidentifying and classifying nuclei, histological stains may be used tomake cells visible. Many dye-based staining systems have been developed,including periodic acid-Schiff reaction, Masson's trichrome, nissl andmethylene blue, and Hematoxylin and Eosin (H&E). For medical diagnosis,H&E is a widely used dye based method, with hematoxylin staining cellnuclei blue, eosin staining cytoplasm and extracellular matrix pink, andother tissue regions taking on variations of these colors. In manycases, however, H&E-stained histologic preparations do not providesufficient information for a pathologist to visually identify biomarkersthat can aid diagnosis or guide treatment. In this situation, techniquessuch as immunohistochemistry (IHC), immunofluorescence, in situhybridization (ISH), or fluorescence in situ hybridization (FISH), maybe used. IHC and immunofluorescence involve, for example, usingantibodies that bind to specific antigens in tissues enabling the visualdetection of cells expressing specific proteins of interest, which canreveal biomarkers that are not reliably identifiable to trainedpathologists based on the analysis of H&E stained slides. ISH and FISHmay be employed to assess the number of copies of genes or the abundanceof specific RNA molecules, depending on the type of probes employed(e.g. DNA probes for gene copy number and RNA probes for the assessmentof RNA expression). If these methods also fail to provide sufficientinformation to detect some biomarkers, genetic testing of the tissue maybe used to confirm if a biomarker is present (e.g., overexpression of aspecific protein or gene product in a tumor, amplification of a givengene in a cancer).

A digitized image may be prepared to show a stained microscope slide,which may allow a pathologist to manually view the image on a slide andestimate a number of stained abnormal cells in the image. However, thisprocess may be time consuming and may lead to errors in identifyingabnormalities because some abnormalities are difficult to detect.Computational processes and devices may be used to assist pathologistsin detecting abnormalities that may otherwise be difficult to detect.For example, AI may be used to predict biomarkers (such as theoverexpression of a protein and/or gene product, amplification, ormutations of specific genes) from salient regions within digital imagesof tissues stained using H&E and other dye-based methods. The images ofthe tissues could be whole slide images (WSI), images of tissue coreswithin microarrays or selected areas of interest within a tissuesection. Using staining methods like H&E, these biomarkers may bedifficult for humans to visually detect or quantify without the aid ofadditional testing. Using AI to infer these biomarkers from digitalimages of tissues has the potential to improve patient care, while alsobeing faster and less expensive. Computational pathology processes anddevices may be used to assist pathologists in detecting abnormalities,such as floaters, that may otherwise be difficult to detect. Forexample, AI may be used to predict the presence of floaters fromindividual regions within digital images of prepared tissue samples. Theimages of the tissues could be whole slide images (WSI), images oftissue cores within microarrays or selected areas of interest within atissue section. Using staining methods like H&E, these biomarkers may bedifficult for humans to visually detect or quantify without the use ofadditional testing. Using AI to infer these biomarkers from digitalimages of tissues has the potential to improve patient care, while alsobeing faster and less expensive.

As described above, computational pathology processes and devices of thepresent disclosure may provide an integrated platform allowing a fullyautomated process including data ingestion, processing and viewing ofdigital pathology images via a web-browser or other user interface,while integrating with a laboratory information system (LIS). Techniquesdiscussed herein may be performed end-to-end using AI techniques, withno or minimal user intervention. Further, clinical information may beaggregated using cloud-based data analysis of patient data. The data maycome from hospitals, clinics, field researchers, etc., and may beanalyzed by machine learning, computer vision, natural languageprocessing, and/or statistical algorithms to do real-time monitoring andforecasting of health patterns at multiple geographic specificitylevels.

The digital pathology images described above may be stored with tagsand/or labels pertaining to the properties of the specimen or image ofthe digital pathology image, and such tags/labels may be incorrect orincomplete. Accordingly, the present disclosure is directed to systemsand methods for identifying or verifying correct properties (e.g.,pertaining to a specimen type) of a digital pathology image. Inparticular, the disclosed systems and methods may automatically predictthe specimen or image properties of a digital pathology image, withoutrelying on the stored tags/labels. Further, the present disclosure isdirected to systems and methods for quickly and correctly identifyingand/or verifying a specimen type of a digital pathology image, or anyinformation related to a digital pathology image, without necessarilyaccessing an LIS or analogous information database. One embodiment ofthe present disclosure may include a system trained to identify variousproperties of a digital pathology image, based on datasets of priordigital pathology images. The trained system may provide aclassification for a specimen shown in a digital pathology image. Theclassification may help to provide treatment or diagnosis prediction(s)for a patient associated with the specimen.

This disclosure includes one or more embodiments of a specimenclassification tool. The input to the tool may include a digitalpathology image and any relevant additional inputs. Outputs of the toolmay include global and/or local information about the specimen. Aspecimen may include a biopsy or surgical resection specimen.

Exemplary global outputs of the disclosed tool(s) may containinformation about an entire image, e.g., the specimen type, the overallquality of the cut of the specimen, the overall quality of the glasspathology slide itself, and/or tissue morphology characteristics.Exemplary local outputs may indicate information in specific regions ofan image, e.g., a particular image region may be classified as havingblur or a crack in the slide. The present disclosure includesembodiments for both developing and using the disclosed specimenclassification tool(s), as described in further detail below.

In the field of computational pathology, information security and dataprivacy should be considered a priority when examining tissue specimen.Personal data and health-related information should be protected. Thebelow description outlines an approach to protecting sensitive andlegally protected information in the development and delivery ofcomputational pathology services and products. Further, the presentdescription outlines a security and privacy by design approach as wellas production system protections for all data, including the databelonging to a medical practice, patients, and/or customers.

One embodiment of a computational pathology process or device may be anintegrated computing platform offering any or all of the following:

-   -   An AI-native digital pathology slides viewer. The viewer may        comprise an interactive user interface or notification        dashboard. The viewer may further support the collaboration and        sharing of slide images.    -   A suite of AI products, including products designed for        different parts of the body (e.g., prostate), which may plug        into different workflow steps.    -   A data ingestion appliance to facilitate the transfer of digital        pathology slides.

FIG. 1 depicts an exemplary global architecture of a platform forprocessing digital slides consistent with the present disclosure. One ormore embodiments may use a cloud provider such as anInfrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS)provider to provide a scalable, reliable, secure, available service tocustomers on a global scale. The exemplary global architecture 100 mayinclude use by a provider in a region 101, and may send digital slidesand images to other regions, such as the European Union or the UnitedStates. The global architecture is developed to account for compliance,reliability, security and privacy across any region.

One or more embodiments may provide any or all of the followingfeatures:

a. Automatically and securely ingest scanned images deposited by WSIscanners 104 into storage 103 and copy these images into secure cloudstorage (e.g., a fully automated process).

b. Leverage the latest advances in cloud computing to automaticallyperform AI computations at scale once the images are received.

c. Provide a state-of-the-art experience to view the uploaded images andpredictions generated by AI products.

d. Maintain strict enforcement of regulatory requirements for ProtectedHealth Information (PHI), including the prevention of patientinformation from leaving its originating geography.

The product architecture is developed to account for compliance,reliability, security and privacy, in accordance with the following:

-   -   Protected Health Information (PHI) may be held in the        originating region: As illustrated in FIG. 1, identifiable        patient information may be kept in the region where the practice        is located. To achieve this, all scanned images may be kept in        an encrypted bucket 111, which may be a simple storage service        (S3) encrypted bucket, physically located in any of the regions.        According to one or more embodiments, a region may refer to        locations having different sizes and characteristics. For        example, a region may correspond to a country, a city, and/or a        hospital, etc. Any metadata related to patients may also be        encrypted and stored in that region.    -   Well-defined API endpoints for data ingestion: in order to        minimize the exposure of sensitive data, a specific endpoint and        a unique bucket 111 may be provided to limit the number of        firewall rules to be created. This endpoint may be stable to        reduce the risk of service disruption.    -   Leveraging authentication provider(s): one or more embodiments        may integrate with an authentication provider using a protocol        (e.g., Security Assertion Markup Language (SAML) 2.0 protocol)        allowing an IT department to manage credentials and access for        authorized accounts.    -   Customer segregation: uploaded images may be stored in cloud        storage dedicated to an institution to prevent data leakages        between customers. Other data may be multi-tenant and not        necessarily segmented from other customer's data.

Within the provider in region 101 location, the global architecture 100may include a Whole Slide Image (WSI) system 102, where WHIs aredigitized and may be stored locally in a storage 103. WSIs may bescanned by slide scanner 104, and sent from the slide scanner 104 to theslide manager 105. From the WSI system 102, digitized images may be sentto either a data ingestion appliance 106 or a Laboratory InformationSystem (LIS) 107. If the images are sent to LIS 107, a user may benotified by a JSON notification that images are available to be viewedon viewer 108.

If the images are instead sent to data ingestion appliance 106, imagesmay be further sent through Web Application Firewall (WAF) 110 to webservices 112, located outside of the originating region 101 of theslides. From web services 112, images may be sent to a web serviceconsole 113 and then back to the original region 101 for additionalprocessing, review, or authentication by authentication provider 109.Images may also be sent through WAF 110 to a viewer 108 outside of theprovider in region 101.

Alternatively, images may be sent from the data ingestion appliance 106to a unique encrypted bucket 111. The encrypted bucket 111 may bephysically located in a number of different regions, for example in theEU, Brazil or in the US. A region may be refer to geographic,institutional, or geopolitical locations, such as countries, collectionof countries, states, provinces, counties, cities, municipalities,hospitals, offices, etc., having different sizes, characteristics,and/or legal or procedural practices, rules, and/or laws. ProtectedHealth Information (PHI) associated with any of the images or slidesmight be prohibited from traveling between encrypted buckets 111 thatare located in different regions. It may be determined, on acase-by-case basis, if PHI may travel between encrypted buckets 111 thatare located in different regions based on one or more rules, practices,laws, etc., associated with the source region, the destination region,and/or any intermediate transit regions through which the data will passduring transfer. Based on the determination as to whether and which PHIand/or other data must be removed, PHI and/or other data may bepartially or entirely removed at the source region before beingtransferred. If the encrypted bucket 111 is located within the US,images may be sent to and from web services 112, which may further sendimages to another encrypted bucket 114 for holding anonymous results.These anonymized results may be used in other products 115, includingdiagnostic tools aimed at a specific area of the body such as theprostate, etc.

One or more embodiments of the global architecture 100 relate toperforming a backup of customer data on a periodic basis and keeping therecords for a predetermined time period (e.g., six years) in order toprovide disaster recovery capabilities.

According to one or more embodiments, customers may remain the owners oftheir data. Customers' data may be used to improve products and tofurther develop the platform and related products and services.

A cohesive Data Loss Prevention (DLP) solution may be deployed to detectany potential data breaches or leakages, monitor data flows and protectsensitive information (e.g., customer data and intellectual property).

The security standards may align with selected standards (e.g.,HIPAA<ISO 27001, GDPR and HITRUST).

Access to physical offices may be controlled by a key card managementsystem. Key card access may also control entry to restricted officeareas. The logs of the key card system may be maintained in a secure logmanagement tool that is used for monitoring, alerting, and eventcorrelation. Physical offices also have video monitoring at all physicalentry ways.

According to one or more embodiments, source code may be stored securelyin a Version Control System requiring employee authentication andenabling auditability. Code changes may be peer-reviewed for quality andpotential security issues. Further, component and product versioning mayenable full traceability.

In addition to traditional testing, Static Application Security Testing(SAST) and Dynamic Application Security Testing (DAST) may be performedfor all the components forming the final products. For maximumefficiency, these tools may be integrated within the SDLC and within acontinuous integration platform.

Patch Management: A regular patching procedure and schedule may bemaintained. Operating system patching may be conducted monthly for allmanagement systems. Product and third-party software patching may bemanaged throughout the development of the product and deployed with eachrelease. The process may comprise a Quality Management System.

Vulnerability management: Security tools may be utilized for conductingactive security and vulnerability scans of the product and productionenvironment. Identified issues are logged and a risk assessment isperformed. A security team may conduct regular assessments, reviews, andaudits of the environment. Further, the team may track and remediatevulnerability issues.

Malware prevention: Malware prevention may include the use of antivirusand malware detection tools. Network traffic may be controlled andmonitored by a firewall.

FIG. 2 is a flowchart illustrating an exemplary method for use of theglobal platform, according to an exemplary embodiment of the presentdisclosure. For example, an exemplary method 200 (e.g., steps 201-209)may be performed by the global architecture 100 automatically or inresponse to a request from a user.

According to one embodiment, the exemplary method 200 for using anintegrated computing platform to view and transfer digital pathologyslides using AI may include one or more of the following steps. In step201, the method may include receiving, by one or more processorsassociated with a provider in a region of origin, a collection of one ormore digital slides. The collection of slides may be provided by WSIsystem 102, as illustrated in FIG. 1 and described above.

In step 203, the method may include sending the collection of one ormore digital slides to a data ingestion appliance. The data ingestionappliance, such as data ingestion appliance 106, may be local to the WSIsystem 102, as described below, or a standalone product. The dataingestion appliance 106 may output the collection of one or more digitalslides as a collection of images.

In step 205, the method may include filtering the collection of imagesassociated with the one or more digital slides through a WAF, such asWAF 110, wherein the WAF comprises at least one rule.

In step 207, the method may include receiving the collection of imagesinto an encrypted bucket physically located in any region, wherein theencrypted bucket retains any patient information associated with thecollection of images in the region of origin to secure patientinformation.

In step 209, the method may include uploading the collection of imagesto a web service or product for further diagnostic processing.

FIG. 3 is a workflow illustrating an exemplary method for use of theplatform with an AI output, according to an exemplary embodiment of thepresent disclosure. For example, AI may be applied in many aspects of acomplete workflow used by the platform, either automatically or inresponse to a request by a user.

According to one embodiment, the exemplary method 300 for using theplatform with an AI output may include one or more of the followingsteps. In step 301, slides are scanned into the workflow. The slides maybe scanned in by slide scanner 104 and stored in WSI system 102.

In step 303, an AI may be run on the input slides. The AI mayincorporate not only the slide itself, but also associated patient data,such as a genetic profile, patient history, other related slides,radiology data, molecular data, clinical data, etc.

In step 305, an AI output is generated from the slide. The output may becustomizable according to user wishes, and may include but is notlimited to the following examples:

-   -   Case assignment: cases may be assigned to an expert pathologist        based on the AI output, to inform pathologists automatically        about the case, and/or to assign cases to multiple pathologists.    -   Case worklist: cases may be sent to a worklist to organize or        prioritize cases according to urgency, to visualize cases        according to importance, and/or to search cases by a desired        outcome.    -   Case preparation: cases may be prepared to order special stains        needed for a patient, to suggest clinical trials for the        patient, and/or to trigger a rescan of a slide, e.g., based on        poor slide quality.    -   Slide tray: slides may be organized within a case according to        urgency or severity, to visualize slides according to        importance, and/or to search cases by detected outcome.    -   Slide viewer: a slide may be viewed with an overlay of a        graphical AI result (a conventional incorporation of AI), to        summarize an AI result textually, and/or to suggest or to        trigger follow-up studies.    -   Pathology report: a pathology report may be pre-filled with the        AI results of the workflow.

FIGS. 4A-C are exemplary architectures of a data ingestion appliance andthe integration of the data ingestion appliance to the platformarchitecture, according to exemplary embodiments of the presentdisclosure. An exemplary architecture of a data ingestion applianceconsistent with the present disclosure may provide a Data IngestionAppliance able to receive notifications when slides are digitized asimages, then queue and upload the new images. Once ready, the acquiredimages may be encrypted (e.g., using TLS 1.2+) and sent to secure cloudstorage where they will be processed.

The Data Ingestion Appliance may seamlessly integrate with one or morescanners, such as slide scanner 104. Data in transit may be encrypted(e.g., using TLS 1.2+ with AES-256 encryption, via industry standardHTTPS).

The Data Ingestion Appliance may be distributed as an Open VirtualAppliance (OVA) file.

In one exemplary embodiment shown in FIG. 4A, data ingestion maycomprise a WSI system 102, an LIS 107, and a bridge 120. The WSI system102 may further comprise an image management system (IMS) or slidemanager 105, a storage 103, and a slide scanner 104, where allcomponents are able to communicate and send slide images between oneanother.

In FIG. 4A, an integration between the LIS 107 and the slide manager 105has been established. This interface allows for digitized slides to beaccessible from the LIS 107. A bridge 120 may be deployed and configuredto consume all information from the interface built from the slidescanner 104, such as the WSI system 102. The interface may be built inHealth Level 7 (HL7), Fast Healthcare Interoperability Resources (FHIR),databases, Representational state transfer (REST) applicationprogramming interfaces (API), etc. Any supplemental information may beobtained from the LIS 107. From the bridge 120, images and associatedinformation may be sent to a cloud 121.

In FIG. 4B, there is no integration between any part of the WSI system102 and the LIS 107. In this option, the WSI system does not contain aslide manager 105. Here, the bridge 120 is deployed and configured asthe main system to consume digitized images from the slide scanner 104and the storage 103. The bridge 120 may also be used to retrieve patientand case initial metadata from the LIS 107. Bridge 120 may then send anyof this information about the digitized images to the LIS 107, or maysend images and associated information to the cloud 121.

In FIG. 4C, an integration exists between LIS 107 and WSI system 102.Specifically, the integration exists between the LIS 107 and the slidemanager 105, which allows for digitized slides to be accessible from theLIS 107. Through this interface, patient, case and slide information maybe available. The bridge 120 may be deployed and configured to consumeall information from the interface built against the slide scanner 104system. From the bridge 120, images and associated information may besent to cloud 121.

FIGS. 5A-C are exemplary architectures of a LIS and the integration ofthe LIS to the platform architecture or other hospital systems,according to exemplary embodiments of the present disclosure.

As shown in FIG. 5A, the LIS 107 may communicate one-way with a viewer108. Once the viewer 108 is opened, either automatically or in responseto a request from a user, a protocol such as HTTPs may request passingall information to identify a case and a patient. This information maybe sent for verification by the LIS 107 to web product 115. Theinformation may be authenticated to a hospital's authentication provider109 using SAML or another standard for exchanging authentication. Onceauthenticated, the images and any associated AI results may be streamedor displayed on the viewer 108.

As shown in FIG. 5B, the LIS 107 may communicate directly with viewer108. The web product 115 may also communicate directly with LIS 107,establishing bi-directional integration of the LIS 107. Once the viewer108 is opened, either automatically or in response to a request from auser, a protocol such as HTTPs may request passing all information toidentify a case and a patient. This information may be sent forverification by the LIS 107 to web product 115. The information may beauthenticated to a hospital's authentication provider 109 using SAML oranother standard for exchanging authentication. Once authenticated, theimages and any associated AI results may be streamed or displayed on theviewer 108. Additionally, the LIS 107 may pull information from the webproduct 115-provided REST APIs.

As shown in FIG. 5C, the LIS 107 may communicate directly with viewer108. The web product 115 may also communicate with LIS 107 via thebridge 120, establishing bi-directional integration of the LIS 107. Oncethe viewer 108 is opened, either automatically or in response to arequest from a user, a protocol such as HTTPs may request passing allinformation to identify a case and a patient. This information may besent for verification by the LIS 107 to web product 115. The informationmay be authenticated to a hospital's authentication provider 109 usingSAML or another standard for exchanging authentication. Onceauthenticated, the images and any associated AI results may be streamedor displayed on the viewer 108. Additionally, the bridge 120 may be usedfor more sophisticated writing operations to the LI 108 or othersystems, such as electronic medical records (EMR) or hospital system130, over any protocol.

FIG. 6 is an exemplary architecture of a slide viewer, according to anexemplary embodiment of the present disclosure. The viewer may be usedfor in vitro diagnostic use as an aid to the pathologist to review andinterpret digitized images of a pathology specimen or case, which mayinclude protected health information (PHI) about an associated patient.For example, an embodiment of a viewer may comprise an AI-nativeweb-based software product that facilitates improved viewing andnavigating of digitized pathology images of slides. The exemplaryarchitecture may allow a user (e.g., pathologist) to view digitizedslide images or diagnostic cases.

The exemplary architecture 600 may include a number of components in alocal setting as well as a number of components based in a cloudcomputing service, such as cloud 140. Within the local setting, theremay be products such as a web application 131, a worklist 132, and amanager product 133 with a corresponding user, e.g., a pathologist,administrator, etc. Any one of web application 131, worklist 132, and/ormanager product 133 may send slides or other information through slidesubscriber 137 to slide queue 136. Additionally, there may be a slidescanner 104, a file system 134 and a database 138 at an institution Awith virtual machine 171. The virtual machine may include a watcher 135which may retrieve slides from file system 134 or database 138 beforesending slides to a slide queue 136.

In the cloud 140, all images may need to be screened through a WAF 110.Then, slides may be sent to a cloud-based worklist 132, an internalapplication load balancer (ALB) 146 or external ALB 147, or to a webframework 172.

If the images are sent to internal ALB 146, the internal ALB 146 willthen sent images to an institution API 148. In turn, the institution API148 may send images to a SQL instance 149, where they may be stored. Theinstitution API 148 may also send the images to a case API 152. Ifrequested, either automatically or in response to a request from a user,the case API 152 will send images to a cluster region 153, whichcomprises a main database 154 and a regional database 155.

If the images are sent to external ALB 147, they may then be sent on toa data ingestion appliance 106, to a regional topic 174. From theregional topic 174, images may be sent to a case queue 150 and then acase subscriber 151, or to case API 152. As described above, case API152 may send images to a cluster region 153, which comprises a maindatabase 154 and a regional database 155. Alternatively, images may besent from the external ALB 147 directly to the case API 152, or toinstance 149. Further, the external ALB 147 may send images directly tocase queue 150.

External ALB may also send images to worklist 132 or to server 141.Server 141 may comprise web application viewer 142 and file system 143.File system 143 may store slide images on the server 141. The server 141may send images on to web service console 113 after a user isauthenticated, which in turn may send images from the cloud 140 to theactive directory 170.

From the web application viewer 142, images may also be sent to cache144, which stores project and user information along with applicationlogs, or to one or more buckets. For example, one bucket may be theinference result bucket 145, and another may be a bucket 111 associatedwith institution A in region 101. Alternatively, the web applicationviewer may send images back to an internal ALB 146.

As illustrated in FIG. 7, one or more embodiments may provide anarchitecture for computational pathology processes and devices (e.g.,prostate cancer detection). The architecture may be used to apply AI andmachine learning models to images from slides which include specimenstaken from part(s) of the body, which may use initial metadataassociated with the patient, and may generate additional metadata (e.g.,heatmap, points of interest, virtual staining, confidence metrics,disease type data, progression estimates) related to the AI and machinelearning models. A heatmap is only one possible return value of thecomputational pathology processes and devices. While a heatmap isdescribed in detail below, other return values may include an overlayimage, text, or other information.

A heatmap may be a two-dimensional (2D) image that identifies aprobability of cancer for each area of the image from the slide. Forexample, each pixel of the image may be assigned a value between 0 and1, with a higher number corresponding to a higher probability of cancerin that particular area of the image.

In response to a trigger (e.g., files being uploaded or scanned, a userinput, etc.), one or more messages may be enqueued into a notificationservice. The messages may be processed and sent to a classificationqueue and forwarded to a classification worker service. If an erroroccurs and a message is unable to be processed, the message may be sentto a dead letter queue to be later analyzed.

For every message received at the classification worker service, theslide (e.g., prostate slides) may be retrieved and a computation may beperformed. The computation may be performed using a machine learningmodel that is trained to identify the biomarker(s) of interest fromrelevant tissue (e.g., cancer tissue), with irrelevant tissue excludedfrom analysis. After the computation is performed, a heatmap that showsa likelihood of cancer in any part of the image may be created anduploaded (e.g., prostate heatmap). Further, after the computation, theclassification worker service may push a notification back to thenotification service indicating whether the heatmap was prepared orwhether the process failed.

Further, the notification service may, based on the notificationreceived from the classification worker service, send a message to aviewer heatmap queue. The messages may then be processed sequentially orin parallel by the heatmap queue and forwarded to a heatmap workerservice. If an error occurs and a message is unable to processed, themessage may be sent to a dead letter queue to be later analyzed.

For every message received at the heatmap worker service, a heatmap maybe retrieved from a results bucket and a computation may be performed.According to an embodiment, after retrieving the heatmap from theresults bucket, the heatmap worker service may create a zipped bmpheatmap and json metadata and pushing it to the results encryptedbucket. According to another embodiment, the heatmap worker service maysend the heatmap to the results bucket along with a zipped bmp heatmapand json metadata.

One or more embodiment may provide any or all of the following features:

-   -   Encrypt data in-transit (e.g., using TLS 1.2+). This may include        any one or any combination of data being transmitted to its        services, data transmitted within the ecosystem, and data being        transmitted back to customers.    -   Store data at-rest, including PHI (e.g., AES-256 encryption).    -   Encryption keys are stored in a Key Management System (KMS)        (e.g., a secure and resilient service that utilizes hardware        modules built to FIPS 140-2 standards).    -   Enforce full-disk (pre-boot) encryption of any or all devices        where customers' data is treated and received.

Computational pathology detection processes and devices may beprovisioned behind a Web Application Firewall (WAF), which monitorsincoming HTTP traffic and filters unpermitted traffic to protect againstmalicious attacks (e.g., injections, DDOS, etc.).

For an increased level of security, one or more embodiments may separateproduction resources from administrative and development resources. Forexample, granular access controls may be used to prohibit customer datafrom leaving the production enclave.

The architecture 700 of the computational pathology processes anddevices may be based in a cloud 140. As described above, the processesmay begin with a response to a trigger 160, which may send a message toa prediction module 161. The prediction module 161 may send a predictionto a viewer heatmap 164, which may use the prediction to create anadditional heatmap 165 or heatmap 166. Heatmap 166 may be zipped andsent to a results bucket 168, and may additionally fetch a heatmap fromthe results bucket 168.

Prediction module 161 may also send an uploaded slide to a predictionclassification module 162. The prediction classification module 162 mayeither send the slide to an additional classification queue 163 or to aprediction classification 167. From the prediction classification 167,an uploaded heatmap may be sent to results bucket 168. Alternatively,the prediction classification 167 may fetch a prostate slide from slidebucket 169.

FIGS. 8A-C are exemplary displays from a slide viewer.

In FIG. 8A, an exemplary screenshot of one embodiment of the slideviewer as described above is shown. In particular, one embodiment of theviewer may comprise an AI-native web-based software product thatfacilitates improved viewing and navigating of digitized pathologyimages of slides. The viewer may display digital scans of surgicalpathology glass slides prepared from tissue at resolutions up to 400×magnification. These digitized images may then be reviewed andinterpreted by pathologists. FIG. 8A shows an exemplary tissue with aninset image identifying a field of view of interest for furtherinspection and navigation.

In FIG. 8B, a second exemplary screenshot of an embodiment of the slideviewer as described above is shown. The exemplary screenshot provides alocation of one or more points of interest within a tissue, according toan exemplary embodiment of the present disclosure. An AI product of thepresent disclosure may comprise a diagnostic decision support tooldesigned to identify foci that are suspicious for cancer on digitalimages of histopathology slides from biopsies (e.g., prostate needlebiopsies). The tool may be designed for identifying various types ofcancers, or a single type of cancer (e.g., prostate cancer). The toolmay detect a concerning morphology, and provide a notification to directthe pathologist's attention to foci suspicious for cancer.

According to one or more embodiments, there is provided a diagnosticdecision support tool that includes in vitro diagnostic medical devicesoftware, derived from a deterministic deep learning model that has beentrained with digitized biopsy slides (e.g., digitized hematoxylin &eosin (H&E) prostate needle biopsy slides) seen and diagnosed at amedical center (e.g., cancer center).

According to one or more embodiments, there is provided a diagnosticdecision support tool that identifies digitized biopsy images (e.g.,digitized H&E prostate needle biopsy images) that are suspicious forcancer (see disclosure below) to draw the pathologist's attention tosuspicious regions of the slide for review. For each analyzed slide, thetool can (1) identify whether or not the provided image is suspiciousfor cancer; (2) provide the location of one or more points of interestin which nearby tissue has a relatively higher suspicion for cancer thanother locations; and (3) generate an additional image containing thedevices predicted likelihood of cancer across the entire tissue.

FIG. 8C is an exemplary screenshot of a computational pathology devicegenerating an additional image containing the device's predictedlikelihood of cancer across an entire tissue, also known as a heatmap.The heatmap may be one of several return values of the computationalpathology platform, as described in detail above.

As shown in FIG. 9, device 900 may include a central processing unit(CPU) 920. CPU 920 may be any type of processing device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 920 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 920 may be connectedto a data communication infrastructure 910, for example a bus, messagequeue, network, or multi-core message-passing scheme.

Device 900 may also include a main memory 940, for example, randomaccess memory (RAM), and may also include a secondary memory 930.Secondary memory 930, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage may comprise a floppy disk, magnetic tape, optical disk, etc.,which is read by and written to by the removable storage drive. As willbe appreciated by persons skilled in the relevant art, such a removablestorage unit generally includes a computer usable storage medium havingstored therein computer software and/or data.

In alternative implementations, secondary memory 930 may include similarmeans for allowing computer programs or other instructions to be loadedinto device 900. Examples of such means may include a program cartridgeand cartridge interface (such as that found in video game devices), aremovable memory chip (such as an EPROM or PROM) and associated socket,and other removable storage units and interfaces, which allow softwareand data to be transferred from a removable storage unit to device 900.

Device 900 also may include a communications interface (“COM”) 1060.Communications interface 960 allows software and data to be transferredbetween device 900 and external devices. Communications interface 960may include a model, a network interface (such as an Ethernet card), acommunications, a PCMCIA slot and card, or the like. Software and datatransferred via communications interface 960 may in the form of signals,which may be electronic, electromagnetic, optical or other signalscapable of being received by communications interface 960. These signalsmay be provided to communications interface 960 via a communicationspath of device 900, which may be implemented using, for example, wire orcable, fiber optics, a phone line, a cellular phone link, an RF link orother communications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 900 mayalso include input and output ports 950 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules may be implemented in software,hardware or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors, or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples to be considered asexemplary only.

1-20. (canceled)
 21. A method, comprising: storing a whole slide imagein a first encrypted bucket, the whole slide image comprising a firstgeographic region and depicting a medical sample associated with apatient of the first geographic region; applying artificial intelligenceto perform a classification of the whole slide image, the classificationdetermining whether portions of the medical sample are healthy ordiseased; generating metadata associated with the whole slide imagebased on the classification; and storing the metadata in a secondencrypted bucket.
 22. The method of claim 21, further comprising:receiving a request to transfer the whole slide image to a secondgeographic region; and removing data from the whole slide image and/orthe metadata based on one or more rules associated with the secondgeographic region to generate a modified whole slide image and modifiedmetadata.
 23. The method of claim 21, further comprising: receiving arequest for the whole slide image from a client device; and in responseto determining that the client device is located in the first geographicregion, providing the whole slide image and the metadata to the clientdevice.
 24. The method of claim 21, further comprising: in response todetermining that a client device is not located in the first geographicregion, processing the whole slide image to determine a second wholeslide image.
 25. The method of claim 21, wherein storing the whole slideimage further comprises performing automatic artificial-intelligencebased ingestion of the whole slide image.
 26. The method of claim 21,wherein applying artificial intelligence to perform the classificationcomprises determining a heatmap based on the classification; and storingthe heatmap in the second encrypted bucket.
 27. The method of claim 21,wherein generating metadata comprises determining a heatmap comprising agraphical prediction of a likelihood of disease in the medical sample.28. The method of claim 21, wherein the classification is performed atleast in part based on patient metadata.
 29. A system for processing anelectronic image corresponding to a specimen, the system comprising: atleast one memory storing instructions; and at least one processorconfigured to execute the instructions to perform operations comprising:storing a whole slide image in a first encrypted bucket, the whole slideimage comprising a first geographic region and depicting a medicalsample associated with a patient of the first geographic region;applying artificial intelligence to perform a classification of thewhole slide image, the classification determining whether portions ofthe medical sample are healthy or diseased; and based on theclassification of the whole slide image, generating metadata associatedwith the whole slide image by determining a heatmap comprising agraphical prediction of a likelihood of disease in the medical sample.30. The system of claim 29, the operations further comprising: receivinga request to transfer the whole slide image to a second geographicregion; and removing data from at least one of the whole slide image andthe metadata based on rules associated with the second geographic regionto generate a modified whole slide image and modified metadata.
 31. Thesystem of claim 29, the operations further comprising: determiningwhether a client device is located in the first geographic region; andin response to determining that the client device is located in thefirst geographic region, providing the whole slide image and themetadata to the client device.
 32. The system of claim 29, theoperations further comprising: determining whether a client device islocated in the first geographic region; and in response to determiningthat the client device is not located in the first geographic region,processing the whole slide image to determine a second whole slideimage, and processing the metadata to produce modified metadata.
 33. Thesystem of claim 29, wherein storing the whole slide image comprisesperforming automatic artificial-intelligence based ingestion of thewhole slide image.
 34. The system of claim 29, further comprising:storing the heatmap in a second encrypted bucket.
 35. The system ofclaim 29, wherein the applying of artificial intelligence to perform theclassification is performed at least in part based on patient metadataassociated with the patient.
 36. A non-transitory computer-readablemedium storing instructions that, when executed by a processor, causethe processor to perform operations comprising: storing a whole slideimage in a first encrypted bucket, the whole slide image comprising afirst geographic region and depicting a medical sample associated with apatient of the first geographic region; applying artificial intelligenceto perform a classification of the whole slide image, the classificationdetermining whether portions of the medical sample are healthy ordiseased; based on the classification of the whole slide image,generating metadata associated with the whole slide image; and storingthe metadata in a second encrypted bucket.
 37. The computer-readablemedium of claim 36, the operations further comprising receiving arequest to transfer the whole slide image to a second geographic region;and removing data from at least one of the whole slide image and themetadata based on one or more rules associated with the secondgeographic region to generate a modified whole slide image and modifiedmetadata.
 38. The computer-readable medium of claim 37, the operationsfurther comprising: providing the modified whole slide image andmodified metadata for transfer to the second geographic region.
 39. Thecomputer-readable medium of claim 36, the operations further comprising:determining whether a client device is located in the first geographicregion; and in response to determining that the client device is locatedin the first geographic region, providing the whole slide image and themetadata to the client device.
 40. The computer-readable medium of claim36, the operations further comprising: determining whether a clientdevice is located in the first geographic region; and in response todetermining that the client device is not located in the firstgeographic region, processing the whole slide image to determine asecond whole slide image, and processing the metadata to producemodified metadata, the processing of the whole slide image and theprocessing of the metadata being based upon one or more rules associatedwith the first geographic region.